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The process o£ research synthesis is examined in this 
paper. The results of case study of what happens to the attitudes 
o£ six experts in the field o£ school desegregation and black 
achievement when they interactively review the relevant research 
literature are presented. The impact of these subjects' resulting 
reviews on the attitudes of a less expert audience (i.e. graduate 
students) is also examined. It is concluded that the 
attitude-relevant aspects of research reviewing are probably best 




case of the research reviewers) there is a high degree of prior 
knowledge. In addition, the diversity of results often found in a set 
of empirical studies will inhibit attitude change in its consumers, 
further impeding consensus. And finally, even when there is agreement 
on the objective outcome <.-;f a set of studies, there may still be 
discrepancies resultin«| from subjective interpretations of what the 
measurements mean. Broader philosophic debates within the social 
sciences which are helping to define the limits of objectivity are 
considered and further studies based on the premise that knowledge 
synthesis is more meaningful if accompanied by descriptions of the 
process that brought the synthesis about are urged. (RDN) 
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Running Head: Six Desegregation Reviews 



Six l)eaeqre«jat» on Reviews: I 

Overv lew 

Invest iqator;i mtereated lm the social paycholotjy ot eUucition 
often confront a time-worn pr-oblem faced by all applied reyearchera : 
how can objective and/or consensual rules of data gathering and 
interpretation be >ised to study issues that are value-laden and tied 
in with well-entrenched attitudes? Pei^aps this problem is best 
exemplified by the debate surroundiny the effects of school 
desegregation. The intensity of social and political pressures, the 
salience of the researcher's personal beliefs, and the complexity of 
the problem, make desegregation ar. especially difficult area for 
interpretation using social psychological principles and techniques. 
This manuscript presents the results of a case study examining what 
happens to the attitudes of six experts on the effects of 
desegregation when they interactively review the relevant research 
lii >rature. The impact of their resulting reviews on the attitudes of 
a less expert audience (i.e., graduate students) is also examined. 

Introduction 

In the summer of 1982, NIE's Desegregation Studies Team (DST) 
undertook an attempt to reconcile the research literature concerning 
the effects of desegregation on black children's achievement. The 
several major reviews in this area had produced conflicting results, 
leading the Desegregation Studies Team to feel there was a need to 
clarify what was kno*m about the effects of desegi-egation, what needs 
to be discovered, and what directions th« next research should take. 

Jeffrey Schneider (1982), the head of the Otisegregation Studies 
Team at the time of this effort, has written: 

DST knew from the beginning of the project that iwkHi of 
the available research on school desegregation and its effect 
on black students' academic achievement ojffered from design 
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tlaws. OST alao knew from the t>eijinninij that much of thii; 
reijearch raised more questions than were answered. We ^Vid not 
know if desegregation research suffera fr<»n this in any greater 
measure than other topical research areas. Nor did we know why 
so many reputable scholars disagreed in their interpretations of 
analytical results. To help answer these questions, NIE 
commissioned a set of papers in an effort to <*tain the views of 
six reputable scholars who had reported opposing conditions in 
this area and one research methodologiat who had not been 
identified as a desegregation researcher. The NIE intercsst in 
this project was in finding if 'inder similar conditions, with the 
same set of data, and common ground rules, similarities and 
differences in scholarly analysis can be i<tentified and 
clarified. The participants were selected by an NIE staff 
analysis of previous research findings, by an informal poll of 
persons er^aged xn desegregation research, and by a request that 
each of the possible participants identify others having similar 
and divergent views from their own. The panel members were 
Robert Grain, Paul Wortman, David Armor, Norman Miller, Walter 
Stephan, and Thomas Cook, who served as metlwdologist. 

The seven scholars met in July 1982 at which time they 
discussed the state of the research literature and agreed on the 
use of a comprehensive criteria in selecting studies to be 
analyzed . . • A total of 157 enfl>irical studies were identified 
that looked at black stuctent academic achievement in desegregated 
schools. The selection process resulted in a "core" of 19 
studies. Panel menAiers agreed, however, tjiat individual 
reviewers would be allowed to add or delete studies fvtm the 
"core" (p- 6-^6}- 

After this initial meeting, the panel members prepared first 
drafts of their papers. They then rec«ivened in December 1982 to 
review and evaluate eacii other's work. 

At the same time that the Desegregation Study Tc«»«s effort was 
beginning, NIE*s Dissemination in Practice Program was agreeing to 
fund my research which proposed to study the literature review as a 
knowledge synthesis process. The first objectives of this research 
were a naturalistic examination of Irow literature reviews are carried 
out and how they are evaluated by interested readers* Obviously, the 
convening of NIE's panel on desegregation and black achievea^t 
provided a rare opportunity for studying the process of research 
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Six Deseyre<jation Reviews: 3 
syntheuis. Six expert rf'seai-c^ieru wer^ aaked to drow concl»mion« 
about a i^imjle hypothesis ummj a nearly cominori set of studies. Both 
the Desegregation Studies Tfeem and the panelists agreed to take part 
xn my research. It is the outcome of this research that will be 
reported in this paper. 

Because of the structure of the panel's assigment, certain 
aspects of the reviewing process could be examined as part of the 
naturalistic study* First, each panelist came to the assignment with 
extensive knowledge of the topic. Nearly all panel members had 
previously written reviews of desegregation research and had taken 
part in analysis of primary desegregation data* Therefore, the sources 
of the experts* predispositions toward the topic could be studied^ 

Second, the assignment of the panel included a f^ase in which 
the quality of dtesegregation research was to be exaanined. This allowed 
for an assessment of the experts' beliefs concerning how a study's 
design affected its informational utility, 

A third set of questions involved attitude change. It was 
possible to compaz^ the panelists* prior beliefs about desegr ion 
with their beliefs at tlw assigpiment's concli^icm* An obvious question 
to ask was: Did the panel experience move tte participants toward 
more positive or negative concli»ions ctmceming the effects of 
desegregati<m? Also, did the {^utmI e)q>erience erdiance or diminish 
participants* confidence in their concli«l<»w? And perhaps most 
ifl^ortant, did the panel experience create greater consensus or 
dispersion ^K»a tte c«>lnions of participants? 

A fourth area of research synthesis that could be examined 
involved the %#ritten products of paMl nmrntoerB. The six written 
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Six Desetjreqation Reviewis: 4 
reviews could be read by an interested audience .irni their reactions 
to the papers aasessed. 

The characteristic of the panel that made asking these qiiestionii 
especially meaningful was that all six reviewers addressed a cominon 
question yoked to a common body of evidence; All reviewers began with 
an identical core of 19 studies but were then allowed to add other 
studies on their own. Xt is important to note, however, that this 
same characteristic precluded the asking of some other questions about 
research synthesis* First, differences in how the reviewers might 
define the notions of "desegregation" and "achievement" could not be 
examined. S<mb panelists found the need for a common definition of 
these terms restricted their analysis of the problem. Also, 
variations in literature searching strategies could not be studied 
because the panel used a pre-existing evidential base. Finally, the 
panelists all agreed that the generation of an average effect size 
estimate and the examination of variance in effect sizes across 
studies was the strategy to be used for integratiwi of study results. 
Thus, while different reviewers rais^t choose different mathen^tical 
formulas , the basic soundness of the quantitative api»roach was 
generally accepted* 

The factors influencing choices of definitions, literature search 
strategies and synthesis techniques are as central to the outcomes of 
reviews as are the factors influencing prcKlispositions , research 
quality judgownts, and r^qport preparation (Cooper» in {Mress) * They 
arc overlo<^(Kl here <mly because the panel was structured in a manner 
that meant these paraawters of integrative research reviewing were not 
as free to vary as were oVhmrn* 
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Collection of Data 

Data collection tor the first part ot thia ntudv was accomplished 
thixjugh two telephone interviews with each panelist. The first 
interview occiirred after the panelists initially met but before actual 
work on their papers began. This phone interview inducted closed- 
ended qiiantitative scale questions, open-ended questions, and 
nondirective requests for general observations. 

Each participant was asked what their predisposition was 
concemir^ the research on desegregation — did it enhance, have no 
effect, or diminish black achievement, or could no conclusion be 
drawn? Participants were also asked how confident they were that their 
interpretation was correct and, in the event that they believed there 
was a desegregation effect, p»rticipants described its magnitude on a 
scale from "very small" to "very large". The panelists next listed 
those variables they felt might mediate the effect of desegregation. 
For instance, among the mediators offered were the child's age at 
desegregation, curriculiaa factors, and st«ff attitudes, to name a few. 

In the final part of the structured interview, participants 
rank ordered six aspects of experimental design with regard to 
their impact on the "informational utility" of a desegregation 
study. The six aspects of experimental design included (a) the 
definition of desegregation employed in the study, (b) the adequacy of 
the control group, (c) the validity of the achievement measure, 
(d> the representativeness of the sample, (e) the representativeness 
of the envirwwental conditions surr««ndinci tte test, and if) the 
statistical analysis. So, for instance, if a participant ranked the 
adeqtiacy of the control group first, it mmant he felt this aspect of 
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Six Desecjregation Reviews: h 
the research dcsiqn hnri the greatest imp^ict, either positive or 
nec|ative. on the value of a study'u results. 

In the second telephone interview, the first three questions were 
repeated, thiis allowing assessments of change in the parwlists 
concliasions, confidence in concl^isions, and estimates of effect size 
magnitucte. Participants wore also asked about their general political 
beliefs and, in open-^nded questions, about their reactions to the 
panel experience. 

In the second phase of the study, the first <ra fts of the 
panelists' written papers were read by 14 post-Masters graduate 
students in psychology and education* The graduate readers took 
part in interviews before and after reading ttie first drafts 
that paralleled the interviews with the participants. Ttie readers 
also completed a separate questionnaire concerning each first draft 
on which they macte jud^ents about the reviewer* s positions and 
the quality of the pap^r* More details on this phase of the study 
will be presented after the results of the interviews with the panel 
mef!:bers are disci;ssed« 

Bemilts; Reviewer Interviews 
Th^ first questions of interest involved the sources of the 
reviewers' predispositions. These in?>ressions were gleari^d primarily 
from the opi\^n-en<ted and nondirective responses of panelists during the 
phone interviews. I will state some of these results as concluoicms 
general to all reviewer when, in fact, they are really hypotheses for 
more broadly basec ami better ccm trolled study. 

The most ii^rtant source of predispositions for panelists was 
the outcomes of their mm primary research, ftamis-on experience with 
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primar/ desegreqation Jata appeare<i to form a central i>et ot 
expectations for the retiulta ot any research on the same topic. 
Seeing — or in this case collecting and analyzing data — is believing. 
Panelists with primary research experience m the area appeared to 
give greatest, and perhaps disproportionate, weight to the outcomes of 

their own studies. 

Because the present study was naturalistic and the data on the 
panelists' initial dispositions was retrospec nive, the assertion that 
outcomes of personal primary research caused dispositions is clearly 
speculative. We must also entertain the notion that initial 
dispositions led panelists to structure their primary research designs 
and analyses in a manner that made supportive results highly likely. 
However, two of the panelists explicitly stated that their beliefs 
about desegregation effects changed in response to data collection. 
Probably the most defensible assertion is that both processes exist in 
nature (that is, primary research influences beliefs and beliefs 
influence research design and analysis). 

Also, there is much evidence that when initial beliefs are vagwe 
or tenuous, the impact of data collection will be great. Personal 
work on primary research is a highly salient event that is 
considerably more vivid than the research of others. The work of 
Tversky and Kalmeman (1973) indicates that such events are not only 
overweighted in judgments but often form a preexisting structure (or 
schema) to *mic*i new information is aasimiXated. 

A second source of predispositicm was th.? disciplinary 
affiliation of the reviewer. Disciplinary affiliations appeard to be 
most ioiportant with regard to the selection of mediators of the 
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Six Descgreqation Reviews: H 
I li*;;efjiM fa on ettect. Kdricatori^s ?5eai-ched mainly tor curriculum 
variables as mediator;^ ot the relation, paychologiats offered mainly 
intervenmy variables associated with interpersonal interaction, and 
sociologists inv<*:ed mainly social structure mediators. Less 
concltmively t disciplinary affiliations also evidenced themselveis in 
predispositions about results. This may occur because of the 
reviewer's general faith in his discipline's level of analysis. To 
use an example other than desegregation , a sociologist might be more 
likely to agree with the staten^nt "social class affects achievement 
motivation" than would a psychologist, who might see social class 
simply as a weak substitute for the "real" psychological detei^ninants 
of achievement motivation, like the aiMunt of independence training 
that goes on in the family. 

Final ly» predispositions appeared to arise from the broader 
political and social belief systems of the panelists. Many topics in 
the social sciences relate to real-world pr<Alems, and certainly 
desegregation is one of these. The outcomes of hypothesis tests have 
implications for the validity of different world views. Cwiservative 
and liberal stances cm general political issiMSS can ''filter" dowi to 
imply particular stances on testable research hypotheses . While the 
present study could only note the consistency, but not causal 
interrelations, of general belief syst^M and specific interpretations 
of eir4)irical data, cognitive consistency theories suggest the 
pressures toward congruence will work to keep ^neral and specific 
beliefs consonant with one another (Abelson et al., 1968). 

The observati<m that pr«lispositions toward review outcomes are 
influenced by personal research, disciplinary affiliation, and general 
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b«liei ivsteirt; is conui:on;;en!nCiil . The point needa to be made here, 
however, becaute it will ishortly become evident that predib-positiona 
explained much of the panelists' reactions to their assignnent. It 
appears that a description of the synthesia process would not be 
complete without clear reference to the role of prior tM«liefs. 

The next set of questions dealt with the impact of research 
deciqn on the utility of a study ''s results. Table I presents the 
pan«linta' riinkinga. The rankings reveaxed neneral agreement that 

Place Table 1 about here 

design factors, associated with internal validity ntoi.t influence a 
study's utility. External validity factors we?^ less important. 
The measurement of the outcome variable, in this case achieverrent , 
revealed much variability in its rated importance. It appeared as 
both the most and least impo'^tant design factor and only two panelists 
anrecd on its rankinn. The statistical «»nalysis used m a study also 
showed variability in ratingi of importance but fhet ratings wei-e 
generally low. Several panelists mentioned that if a study had 
deficiencies in statistical analysis, these could be corrected after 
the fact. 

A Spearman correlation between pairs of rankings revealed an 
average r of *.A7, This number is not dissimilar fro« correlations 
found in broader studies of research quality judgments (Gottfredson, 
1978) . The pair-wise correlations, however, ranged from -.29 to ♦.77. 
This finding led to a testing of whether disagreements about the i»|>act 
of design factors on study utility were associated with disagreements 
about the effect of desegregation. To do this, participants were 
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ronk«.i it\ term;; ot th»»ir f«rceptionu of the et fecti veaeua ot 
Ueiieyreqation. Then, the difterence in rajika between each pair of 
panelists was correlated with the correlation between their x'ankings 
ot the design factors. The resulting Spearman r was -.78, indicating 
that those panelists who disagreed most -about the effects of 
desegregation also disagreed most about relative impact of design 
factors on study utility. Of course, this is. only a crude descriptive 
device and we cannot determine whether diffex'ences in quality criteria 
account for differences in interpretations of desegregation research, 
or vice versa, or whether both beliefs are a function of yet a third 
variable. However, this evidence does support the earlier assertion 
that dispositions toward specific empirical hypotheses cannot be 
understood in isolation of broader cognitive structures. 

The final set of reviewer data involves the changes in panelists' 
attitudes toward desegregation. Before the panel began, three 
participants expi^essed a belief that desegregation had positive 
effects and none changed their mind. Two of three panelists who 
believed <tesegregation sonetinwss had no effect araJ sometimes had 
positive effects experienced no general attitude change. One 
participant changed his opinion from this equivocal position to the 
position that no conclusion could be drawn, due primarily to an 
enhanced appr€K:iation of the complexity of the issue. 

Kith regard to panelists' confidence in their cwKrlusions, 
three participants found the experience enhanced confidence in their 
beliefs. For two participants, this change was dramatic. On a scale 
from zero (not confident at all) to 10 (totally confident), they moved 
from 2 to 6 and frcw 3 to 9. The third panelist showing enhanced 
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confidence im^ved frwn 7 to 9. One partici(>ant lost a 3«all* but 
petxreptible, aaioimt of confidence m his conclusion moving from 3 to 7 
on the scale* Tuo participants reported no changes in confidence 
level remaining at 7 ami 8 on the %ale« 

Next, the panelists' estiiv^ted the magnitude of the desegregation 
effect* Three panelists revised upward their estimates of magnitude 
moving from 'very small" to ••3nall"» from "small to imxterate" to 
"moderate", and from "moderate^* to "moderate to large". One 
panelist's estimate that the effect %#as "very small to small" did not 
change and two participants, who were reluctant to estimate an effect 
magnitude before the panel began, estimated the effect as between 
"very small'* and "?:. ?». ^" when their work was done* These results lead 
to a conclusion that there was some movement towartl more positive 
impressions of t^^ effect of desegregaticm* However, it is important 
to point out that a reading of the panelists* papers, v*ien compared to 
their earlier writings, generally gives the qpposite i^ression — that 
is, desegregation seems less isqpactful in the rwwer works. This 
asses^nient is suf^fKirted by data on tifw graduate student reactors* 
impressions before and after reading the papers, to be described 
stwrtly . 

The c<mf lie ting is^^ressions given by the phone interviews and 
the written psqpers provicte an i^>ortant insight into tlie estimation 
and interpretation of effect sisMS. Cooper (1981) has argued that 
the substantive interpretaticm of an effect ccmtains two co^Hments* 
The first c om po n ent is ttm mathematical estimation of the magnitude 
of the effect itself. this estimate t thm paoielists achieved a 
hi<^ degree of correspondence . All of the panelists agreed that the 
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Six Desegregation Reviews: 12 
eft>ct was positive (i.e., desegregation enhanced black achievement) 
and the estimates ranged fnwi d = .04 to d = .17, a difference of less 
than two months of gain on standard achievement tests. The estimate 
of effect probably captures this mathematical compwient and indicates 
that for half the panel i'sts it was larger after the panel experience 
than before. 

The secwid component of effect size interpretation, however, 
involves the choice of cwitrasting elements for piirposes of comparison. 
That is, effects are rarely interpreted in a vacuum. Instead, they 
are contrasted with effects of a conceptiially similar nature. In this 
case, panelists rai^t ask themselves, "In comparison to other known 
educational interventions, how effective is desegregation?" It is 
this interpretation which appears to have been most prevalent in the 
panelists' written papers and, apparently, several panelists found the 
desegregation etfect did not compare favorably witt other forms of 
educational intervention. Put differently, for some, the panel 
experience may have raised their expectation ccnceming what the 
desegregation effect needed to be in order to be useful. When this 
second component is added to the mattematical evaluati<»i of effects, 
the ^nerally positive pictair« becomes soraewhat i^ative for so^ 
panel mea^ers. 

The estimates of effect also give some clues about whether the 
panel experience created consensus or disperaicm of opinion. Clearly, 
there was as much disagreement among participants when the panel 
concluded as when it began, only after the experience, atme panelists 
more firmly held their beliefs. Ho*#ever, there are some mitigatir^ 
circumstances that offer consolation. First, the issue of 
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However, there are some mitigating circtaastarwes that offer 
consolation. First, the issue of ciesegregation is an exceptionally 
emotional one involving moral and political stances as well ais the 
scientific perspective. Other issues t»y prove more amnable to 
efforts at consensus building throu^^i reseaixh synthesis and the panel 
format. Second, the members of the panel brought an unusually large 
degree of prior experience to their task. Their initial positions 
were well thought out, caq>lex integrations of knowledge acquired over 
years of study. Any expectatiwi of dr9matic attitude change would 
have been unrealistic and contrary to research that indicates prior 
knowledge and experience with an issue makes attitude dniange more 
difficult (e.g., Wood, 1982). 

ConclusitHi: Reviewer Interviews 

In W4m, hours of conversation with the panelists revealed that 
they probably hold disparate views on many social issues — in fact, 
they were chosen for partici|»ti':r» based partly on their different 
perspectives. Hy impressiai was that the enpirical data did create 
convergence in their thiidcing on the effects of desegregation though 
most of this occurred before the panel was c<»ivened, when the 
panelists' attitudes were more malleable. I think the panelists' 
posltiiMis would have been even more diverge had no data or prior 
synthesis activity t^en- place* 

Finally, a potentially encouraging outcome of the panel concerns 
something that did not happen. Recent experimental evidence indicates 
that when people with conflicting beliefs are exposed to a set of 
studies containing conflicting results, attitudes can become even wore 
polarised (Lord, Ross, C Lepper, 1979). Other research indicates that 
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further thougiit about an Xiisue, even in the absence of new 
mtormat'on, leads to polarization m the direction of one's initial 
tendency (Tesser, 1978). There is little evidence that such attitude 
polarization occurred among the paru?lists* Two possible explanations 
for wtiy panelists* beliefs did not stK>w further dispersicm can be 
offered* First, as noted above, the panelists entered this exercise 
with highly refined and rehearsed positions, limiting the opportunity 
for change in any direction. The second explanation is less 
encouragir^* While the panelists began with differing c^yinions on 
desegregation's effectiveness, nwie of the panelists held the belief 
that desegregaticsi had negatiw effects on black achievet^nt. In 
otter words, panelists disagreed about the existence or magnitude of 
positive effects while agreeing that the effect was not negative. The 
Lord et al. (1979) research demonstrated polarization sm>ng perswis 
who believed the same interventicwi had opposite effects- It is 
impossible to tell whether or not the inclusion of panel a«mibers who 
believed ctesegregation had negative effects would have revealed 
evidetKre for polarization (by, say, the msgative groijqp lna:^asing its 
estimate of the negative effect size or showing enhanced confidence in 
their beliefs paralleling the aiai^s in cwifidence demonstrated by 
the positive panelists) « 

Itesults: Reacter Reactions 
tteasures of change . The second phase of this study involved 
obtaining reader reactions to the six reviews. First, the fourteen 
graduate student readers were intervieiMd before r<»adir^ Vtm rev lews » 
Several quest! cms cofK:erned the readers' educational and tc^ical 
background and general political beliefs. Four questions were 
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identical to the repeated measurements obtaitYed from the panelists and 
were meaitt to gauge the readers' beliefs about the effectiveness and 
mediators of desegregati<m and their confidence in these jud^ents. 
Also similar to the panelists" . these four qtiestions were 
readministered after all six reviews had been read. The cAiange m 
responses to these questions will be ex^incKi first. 

The grsKkiate readers' general beliefs about desegregation's 
effectiveness changed little as a function of readii^ the papers. 
Eight readers who initially felt desegregation enhanced black 
achievement did not change their belief, as did one reader who felt 
desegregation had no effect. Three readers who initially felt no 
conclusion could be drawn changed to an opinion that the effect was 
positive while two readers showed change from the latter to the former 
position. In contrast, the graduate reader's beliefs about 
desegregation as measured by three other indices changed significantly. 
Table 2 presents the readers' (a) estimated effect size, (b) confidence 
in beliefs, and (c) ni»dber of suggested mediators of the effect. 

Plao0 Table 2 abmit here 

In general , the reader's perception of the positive effect of 
desegregation dropped |MwipitousXy from before to after reading the 
reviews (t(12) » -3.97, p<.0(K). Before reading the reviews, the 
average effect size estimate was about "moderate" , **iereaa afterward 
it was less than ••small". Eight of 10 readers who estimated the 
effect aiz9 both tines revised their eaUnates domard. 
Interestingly « the reader's average magnitude of effect was 
considerably greater than the reviewers' eatiwate before reading the 
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papers (A. 41 verisus 1.00, resf^ctively ) but was smaller afterward 
(2.*j8 versus 3.1b). This reinforces the impression that the papers 
ffiay have conveyed a less positive evaluation of the desegregation 
effect than did the reviewer's responses to the interview question. 

With regard to the variety of reader opinions, it appears that 
reading the reviews created sowe convergence of beliefs — the standard 
(teviatlon in effect estimates was about two and a half before and 
one and a third after reading the reviews. However, this finding 
needs cautioi^ interpretation because two realtors %«ho estii«ated 
"small" (3) and "moderate" (5) initial effect sizes did not offer an 
estimate after reading the reviews while two readers who initially 
left this question blank estimated "very small" (1) effects after 
reading the papers. 

The graduate students' average level of confidence in their 
beliefs before reading the papers was identical to the reviews' 
initial confidence levels (6.00). Ttiis would indicate t*iat swbjective 
confidence in beliefs about an eayiirical research area is not directly 
related (if at all) to objective expertise on the topic. However, 
after reading the pliers, the stuHlents' confidmntx level tttvSed to be 
hi^er (7.00; ♦ (14) - 1.66, p<.12) but tJw level did not Jump as much 
as did the reviewer' confidence (7.67). 

Finally, remUng tt» reviews led grsKliMte stttdents to cite more 
mediators of the effect of desegregation than they initially proposed, 
though statistically the effect only a{^»roac»^ significance 
(t(13) - -2.06, p<.07). While abckit six and a half mediators were 
mentioned on average before reading the papers, almost nine were 
mentioned after reading the papers. Nine remdera enhanced the 
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con^^lexlty of their beliefs about deseqregation'i? effect* 

It is iiq>ortant not only to note that the readers of feared more 
mediators of dtesegrc^ati cm effects as a function of reading the |>2^^era9 
but also to know what kinds of mediators showed the greatest irKnrease 
in number of citations. In ordter to find this outt a c<mtmit analysis 
was condiKited <m the open-ended r9sp<mses of readers to the qiaestion 
''What variables or conditi<His would yoii suggest ^at mi^t s^diate 
the effect of desegregaticm on black achievement?'^ 

Figure 1 lists the €0 serrate categories into i^idi mediators 
were placed. The 60 categories wex^ then redi)K»d to II broader 
categories for statistical analysis. The 11 categories of n^iators 

Place Figure 1 about here 

were: the attitudes of the people involved in the desegregation 
effort, including parents, students, teachers , and sc±tool 
administrators; the family and personal backgrowd of the students; 
the ^ographic and cultural backgrowid of the community; the 
characteristics of the ^^wol, of classrooms, and of teachers; the 
measurament mi type of acdiievment examined in tlM research; 
cmiditicms under Mtii^ desegregation was acco«4>lishedi the emnvit of 
resources available to a scAkk)! district; md the nimber of and 
reapanae to prcblem asaociated tfith c^ianging al^identa f ron om sc^ol 
to another. An eleventh, niSMllaneous categcK^ waa used for nediatora 
which coders felt did not fit into any of the previous 10 categories. 

The categories were developed by the principal investigator based 
on a reading of ^achiate students' respof^M. Tluin, e«:h reactors* 
responses were coded by two other Judges. Their inter Judge reliability 
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across the II cateqoriea, as meaaijred by Gotten 'a Kappa, waa , 7e* when 
they coded the reader's prior beliefs and . 7fa wtien they coded the 
graduate students' respcMfuses to the ciuestion after reading the reviews. 

Table 3 presents the results of this analysis. First, it should 
be noted that both prior to reading the reviews and after reading ttie 
reviews, the graduate stuidents cited the attitudes of those involved 

Place Table 3 about here 

in the desegregation effort as the most important influence on its 
effect. Vrm two types of mediators that evidenced significant 
increases in citation frow before to after reading the reviews were 
the tw categories involving how achievement was defined and how 
desegregati<m was accoiBplished. 

Reading the reviews led the graduate students to place an 
increased emphasis on factors such as how achievement was measured 
and interpreted, **iat subject matter was under scrutiny, **»ether or 
not the desegregation effort was voluntary or forced , how involved 
the coMunity tias in its iii9>lementati<m, tiw rraed or distance of 
busing, the black>white ratio created by desegregation, the length 
of tine betwven ia^»lem»ntation and the asaessMimt of outcome, and 
finally whether it was blade «r white children who bused. There 
was also soma indication that the veadmvn placed greater c^>hasis on 
the need for planning and the avoidance of disruption after the reviews 
were read. 

The next wMlysis involved relating the readers* iMckground to 
their beliefs about iteaegregation before ami after reading Vhm revie««s. 
These results are displayed in Table 4. Table 4 contains nine 
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variables. The variable "topic fawiliarity" is a composite of the 
grodiiate stu<ients' year in graduate school and their responses to the 

Place Table ^ about here 

questions, "How familiar are you with desegregaticwr. research?" and 
"How many scholarly articles related to desegregation have you read?" 
The intercorrelations among these Iteots were .75, .01, and .83. The 
three items »«re wly weatftly related to the graduate students' answers 
to the question, "How familiar are you with research methodology?" 
(correlations ranged from r = -.10 to r = .02) , so this response was 
left as a separate variable, as was the questiwi asking reatters to 
place themselves cwi the political spectrum. The last six variables on 
Table 4 should already be frailiar. They are the graduate students' 
assessments of the direction and magnitude of the desegregation effect 
their confidence in their beliefs, and the number of mediators they 
cited before and after reading the revievna. 

In examining Table it ^uld first be noted that the 
correlations are based on between only 9 and 13 degrees of freedom. 
Therefore, the testa have wry poor statistical power and in Bcme 
instsnces X will mention correlations that do not reach traditional 
levels of significance but are large enough to warrant attention in 
future studies. 

With regard to the relationships between the readers* politics 
and Uieir desegregation beliefs, no correlation between political 
beliefs and other variables was found to be significant or to produce 
a trend. The strongest relationships were between political beliefs 
and the aise of the <te8egregation effect estimated after reading 1^ 
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papei^o Specifically, the congelation of r - .39 indicates that wore 
liberal readers estxfl»teri larger effects after z*eadit^ the reviews. 
Before reading the reviews, tkw iNslaticm between political beliefs 
and effect size was near zero (r » «0l). Althouigh both of these 
figures are n<msignxf leant, the directimi of ch^unge in the relation ^ 
is revealing* If the data were building consensi^s by replacing 
abstract beliefs with objective outcomes, %#e would have expected the 
opposite pattern of results, that is, a disappearance of relation 
between broad political terwsts and beliefs about desc^gregatxon effects 
as a fimction of re^dlr^ e^lrical reviews* Instep, something like a 
polarization effect €^pears to have occurred in that the readetn^' 
political beliefs may have guicted ttieir processing and intexpretation 
of the data. Also, in future atudies of this issue, it would be wis^^ 
for researchers to ei^ure a wider range in initial political beliefs — 
in the {nr^esent study reactera predooiinmitly described themselves as 
liberal. A less restricted range of beliefs mig^t reveal a larger 
initial relationship. 

The research e3q»rtise of the readera was negatively related to 
their effect Bizm cmtimates (r m ^.45) and positively related to the 
nuntoer of mediators they cited (r « ♦.54) after reading the reviews. 
Thus, it ai^pears that grac^te sl^Ktents wit^ loore research experience 
came to see a more equivocal and coa4>lex situation as a fuMticm of 
the information input. This is jw^ob^ly what we wcmld expect one of 
the iiq>licati<ms of research e^qpertise to be* 

Finally, familiarity with the topic of desegregation ims 
positively related to ttm graduate students' confidence in their 
beliefs both before (r - .48) and after (r » .38) reading the reviews. 
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but the relation was sommihat stronger at the f irat measurement. 
Possibly, the mitigated effect Is a function of those gradiiate st»tUents 
who were unfamiliar with the topic showing increased confidence in 
their beliefs after reading the six reviews Topic familiarity was 
also fwgatively relatec* to effect size estimates before reading ttw 
revietfs (r = -.47). This indicates that ttwse readers -least familiar 
with desegregation research had the higher expectations for 
desegregation's effect. Interestingly, after reading the reviews, 
topic familiarity was positively related to effect size estimates 
(r = .35). An examination of the raw data revealed that these readers 
who were most ijnfaiailiar with desegregation research had their beliefs 
about effectiveness take the more precipitous drop fr«B before to after 
reading the reviews, i^iain, ttiis substantiates the finding ttiat 
faniliarity with a topic is positively related to resistance to dhange. 

Evaluations of individual reviews . The next set of analyses 
involved the graduate readers' evaluations of the Individual reviews. 
One question asked the reader to stinate the reviewer's position on 
the direction of the desegregation effect; another question asked what 
the reviewer would estimate desegregation's effect size to be; and a 
third question a^ed the reader to list the reviewer's aiggested 
influences <m in^pact of desegrogatiwn. Readers were also asked to 
gauge their own ccwif idence in how accurate their interpretatlOT of the 
reviewer's position was. The second set of questions a^ed the reader 
to make sevrn evaluative judgRttnts about the review, thmaet ji^^jments 
included how clearly the probleai was defined in the review, how 
exhaustive was the research covered in the review, how twll the 
reviewer evaluated the strengtha and weaknesaes of desegregation 
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research, how well the? revie%#er synthesized the separate studies into 
a coherent whole, and how clearly the review was written. Readers 
also gave an overall judgment of the review's quality ai.d 
pei^isuasiveness. Finally, reacters were asked to make open-ended 
cc^aronts ccmcemxng their evaluaticm of the reviews; 

The first analysis involved an informal ccmtent analysis of the 
open-ended comments offered by readex^ at the hottoiQ of each evaluation 
sheet. This was undertaken to ^t some id€» of the subjective 
dimensions readers eo^loyed to evaluate the reviews, rather than those 
offered on thei.i<iuestionnaire itself « 

The readers most often mentioned that a fmper was eittwr well or 
poorly organized. Seccmd fl«>st frequently mentioned was writing style, 
in particular the ^thor's ability or inability to kecqp the interest 
of the r€»der. Third was how %#ell or poorly focused the paper was on 
the topic of interest. Fourth was how well or poorly the reviewers 
used citations to substantiate any claims made in their paper. Next 
was attentiwi or inattenticm to variable definitions and to nmdiating 
influences. Also mentioned were how well or poorly the reviewer 
described the methods of the individual ctesegregation studies wkI the 
met>»ds of the review itself. Finally, the manuscript preparation, 
typically involving negative comments alMut typos or missing tables , 
was also nmtic^ned by several of thm readers. 

To exMirw t»w responses to thm closed-endad parts of the 
questionnaires , a factor analysis was first performed on the five 
judgments concerning specific qualities of the reviews. The factor 
analyses were performed for each review separately. They revealed 
that a single quality factor probably underlied al) five judgments. 
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The first principle compcxieiit irt each analyjua accoimted for between 
58% and 83% of the variaiK» m the five acorea and other factors had 
high loadings by only »lngle variables. When the first five questions 
were standardized and c<^ined into a composite measure of review 
quality, this composite correlated with the reviewer's overall judgment 
of quality ranging from .84 to .96 across the six reviews. There fr re, 
the composite measure of quality based on the first five qijestions 
were used in all subsequent analyses rather than the single seasure. 

The final analysis entailed an examination of the covariation 
between readers' evaluations of the quality and persuasiveness of a 
review and their perceptions of the reviewers' beliefs. To carry out 
this analysis, the readers' responses to the closed-ended items on 
the individual review questionnaires were correlatd with one another 
as well as with several of the reviewers' answers to questions during 
the telephone interviews. These correlations were computed for each 
reader separately, the correlations for the 14 readers were ttien 
transforoied to Z-^orea and the Z-acores then ent;ered into a cme-sa«ple 
t-test to determine if they were significantly different from zero. 
An example will clarify this procedure. Each reader made six quality 
judgments, one for each review, and six judgments of the persuasiveness 
of the reviem. Therefore, for earf» reader a correlation was conputed, 
based on six paired observations, which described the direction mkJ 
strength of the relation betMsen quality and persuasiveness Judgmmits 
for that reader. This jprooeduro was carried out for each reader and 
then the set of 14 correlations were transformed to Z-acor«a to 
normalize their distribution. Finally, by testing whether the average 
Z>score was significantly different from zero, it could be established 
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whether or not a relation between the two judcjinenta cotiin be interred 
troro the aample ot readers. 

Figure 2 displays the significant relations between the readers' 
perceptions of the individual reviews and selected reviewer 
characteristics. In interpreting these relations, it must be kept 

Place Figure 2 about here 

in mind that they ax*e associational in nature, so inferences about 
causal direction and nonspuriouaneSs imjst be trade with extreme caution. 

Of foremost interest are those percept lo. a of reviews that 
correlated with judgments of quality and persuasiveness. It was found 
that the quality of reviews was not related to the substantive petition 
of the reviewer, such as the perceived effect size or number of 
mediators offered. Again, we roust bear in mind, however, that tte 
sample of both panelists and reactors were all on the same side of the 
issue (i.e., no one felt desegregation had negative effects). Had a 
wider range of beliefs been represented, a relation between judged 
quality and substmtive posititm ni^^t have energed. Instead, reviews 
judged to be of higher quality were those whic^ the readers felt the 
inost confidence in the accuracy of their interpretation of the 
reviewers' position* regardless of what that position was. 

The persuasiveness of a review was positively correlated with 
its quality, with the reader's confidence in its interpretation and 
with the number of mediators the reader thoiKjht it mentioned. This 
last relation, indicating more persuasive reviews were those seen as 
mentioning more mediators, is of special interest because it indicates 
a relati<m between persuasivmess and complexity. It would be 
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important to k-.ow if thi-j relation is general across all topics or 
if it holds only for topics where reactors are predisposed to believe 
the relation under study is very complex. 

The magnitude of the deewgregation effect a reader thou^t a 
review espoused was positively related to ttie reviewers' stated effect 
size and to tlie liberalrwss of the reviewers' stated political beliefs. 
Effect size perceptions were also negatively related to the reviewers' 
stated confidence in their interpretation and to the nuaber of 
mediators a reader thought the review mentioned. Finally, the number 
of mediators a reader thought a review mentioned was positively 
associated with the reviewers' ctsrtfidence in their interpretati<Mi and 
with the reviewers' politics. 

To suraaarize the results of the sec<md phase of this study, the 
graduate readers showed greater flexibility in their attitudes than 
the rev -ewers, due undoubtedly to differences in the two groups' 
initial states of knowledge. In general, readers' beliefs became more 
congruent with the reviewers' beliefs snA more complex as a fiaK:tion 
of reading the papers. Wwt was not examined, however, is whether 
under natural circumstances readers would have chosen to eiqpose 
themselves to the variety of opinions contained In these papers or 
whether they would tmly seek out papers that would confirm m* bolster 
their initial positicms. This latter f^iemxnemm, called selective 
exposure, is a source of controversy in social psychological research. 

The analysis of readers' backgrounds found little initial bias 
due to political beliefs but some indication that increased knowledge 
of the topic also led to increased congruence between general political 
beliefs and beliefs about desegregation. This may indicate the 
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presence of the phenowenon iRentxoned earlier--that when people are 
confronted with a smorgasbord of empirical findings they give greatest 
weight to t^se that are consistent with prior beliefs. 

With regard to the evaluation of individual reviews, these 
appeared to be relatively free from the influence of initial 
topical beliefs. Quality judgpnents covaried with confidence in 
interpretation. Judgments of the persuasiveness of a review covaried 
with quality, ccmfidence in interpretation, and ntwiber of mediators 
nentioned. 

CotKilusions 

The basic principles of attitude change appear to af^ly to the 
process of eiqairical Knowledge synthesis. Rather than being an 
activity of a qualitatively different order, the attitude-relevant 
aspects of research reviewing, as exen^lified by NIE's paiwl, are 
probaoly best understood by noting the special characteristics of 
the people and circumstances involved and by applying established 
principles to t^is unique situation. 

First, the desegregation panelists began their synthesis task 
with a great deal of prior knowledge, aa do nost research reviewers. 
This knowledge is not only great in Magnitude but it is also well 
organized in a complex structure. These characteristics of reviewers 
will lessen the possibility of change in basic beliefs because the 
synthesizer will (a) encounter few arguments which are truly novel 
and (b) have a co^itlve schema with whi^i to Integrate or counterargue 
inforaaticm that is new. 

Readers of research review *iho brii^ lees tenwious beliefs to 
the t<9ic area, m»ch as t*»e grmdMate students in ttiim study, are ■ore 
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likely to oxperisnce attitude change and eiriianced con^plexity of 
beliefs. The amount and direction of this cHange apparently will not 
only be a function of the reviewer's concltai<ms and treatment of the 
relevant material, but also of the effectiveness of the reviewer's 
presentation, in particular the <»*ganization «kI style of the 
manuscript. 

Aside from the characteristics of the actors in the review 
process, the empirical <^aracter of researdi syntheses, like the 
question of the effects of ctesegregaticm, is also critical to 
understanding related attitude change. Scientific studies claim 
a certain degree of cAjectivity for their remits, aKi bjr ii^lication 
assume that identical tests of the sane hypothesis will lead to 
identical, or at least similar, results. Yet, sets of ei^irlcal 
studies yield varying, often conflicting findings. This may be 
because methods are different, hypotheses really are not commensurate, 
or the assumption of objectivity is false at the start. 

The di^nsrsity of resailta oftm fouul in a set of e^irical 
studies will inhibit attitude change in its consumers. If an initial 
opinion is minimally reastmable, an examiner of the related research 
will find some studies that confirm the initial belief. Obviously, 
the diversity in results will also greatly impede the ability of 
research to create consensus among rwiwwa or review remlera. In 
the instance of less knowledgeable conmMmm, like the graduate 
readers in this study, the diversity of results «ay lead to the 
formation of attitudes consistent with other aore general ly held 
beliefs (e.g., political philosophy). 

Finally, even when a certain degree of oonsenous is reached on 
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the objective outcome of a s^t of studies, in this cause the size of 
the desegregation effect, the varying perspectives of reviewers and 
readers can still create discrepancies in the subjective utilities 
that arc used to interpret the findings. Thus, i*iile a great deal 
of agreement mi^t be reached on the observation that an eight-ounce 
glass contains four ounces of water. Uiere can still be naich 
disagreement about wheUier the glass is half empty or half full. 

It ia this last point which speaks most directly to studying the 
social psychology of education in general. Many problems in education 
require both objective measurements and subjective interpretations of 
what the measurements mean. Questions like "Is a teaching method that 
fosters independent learning better than one that maximizes 
achievement?" or "Is the cost of a program justified by its academic 
outcomes?** are rust anst^rable with data alone. Researchers cannot 
resolve these debates — they can only help vhe <tebatex*s ground their 
arguments in more precise and reliable information (e.g. , How much 
independence ia gained by a teaching ii«thod? How much added knowledge 
does a program buy?). Mhen the researcher moves fro® informing 
debaters to joining ttie debate, they move from the role of researcher 
to that of advocate. 

Of cmrse, the distinctitm between these roles is not as clear as 
it sounds. Often the choice of isroblems and how they are to be 
studied can be as value-ladom as how data are interpreted. For 
instance, in assessing the effects of desegregation, its impact on 
white as well as black student's achievement, or on housing patterns 
aight lead to different conclusions about the treatment's 
effectivmoss. Thus, social psychologists of education will be 
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challenged at every atatje of a research endeavor to remain aware of 
when they are acting as knowle<i^-seekers and when as advocates. 

If this assessment of the research synthesis process in 
educational research seena di^artening it is only because the 
expectations for modem-day social science nay be too hi^. It is 
unreasonable to cwisider a failure anything less than complete 
agreement mong scholars. It is also unreasonable to overlook those 
aspects of a problem up<m whi<*» agre«Bent has been reactied: NIE's 
panelists achieved rei^urkable c<»isensus on the magnitucte of the 
desegregation effect. 

Also, it should be remen*ered that the devol<^ent of new 
techniques for increasing the objectivity of research synthesis, such 
as computerized literature searching and meta-analysis, is occupying 
the time of many social science roe thodo legists. Using these 
techniques, as the panelists did, probably led them to greater 
agreement than would have been ttm case otherwise. It is likely that 
further refinements in method will l^ad to more consensus a»Kl. just 
as important, better understanding of why disagreement occurs. 

Finally, philomiphic drbatea within the social sciences are 
helping to define the limits of objectivity, and as such will help 
us know when a research synthesizer is talking as a scientist and when 
as a citizen. 

In sum, th« convening of HIE's panel on dssegrvgation mvi bla«* 
achievement proved to be an ei^iting natural laboratory for the study 
of knowledge synthesis. It WMld serve the purposes of both the social 
science commmity and the general public if efforts of this sort were 
continued in the future. Social scientists would learn about basic 
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social psychological processes ?nd how they mf liienc;e their own 
activities. The general public would learn more about what to expect 
from social science products and how best to interpret them. Knowle<^e 
synthesis is more meaningful if it is accompanied by descriptions of 
the process t*»at brou^t the syntiiesia about. 
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Table I 

lapaet of R«aearcb Design Factors on the UtlXlty of a Study* a Results 

Bxiwrimta Ifenipulation 1 1 3 2 1 2 1.6 
(Dieflnltlon of Desegregation) 

Experimental Conparison 3 2 2 1 2 1 1.8 
(Adequacy of Control Group) 



2 6 1 3 6 3.6 

relation Generality 5 5 »• ^ 5 5 



Outeom Measure 
(Neasiuwsent of AehieTeront) 



SQOloi^c«l Generality 
Statistical Analysis 
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T«bla 2 

Gsitduata ReiUler Ballefe About Dcaegrcgatlon Befora and After Raading 



<)«aatioBt 



Befora 



Aftar 



Vhae la tfca alca of tKa deaegregatlfm affect? 



4.41 
(2,57) 



2.S8 
(1.38) 



How conf idant ara you that your belief le accurate? 



6,00 
(1.92) 



7.00 
(1.36) 



llhac wliAlea of condltlona aedlata the effect? 
(tti»bar aantioiicd) 



6.46 
(3.45) 



8.93 
(5.7<^ 



Motai Standard devlktioaa ara to parantheaea . 
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Table 3 

Ownge In ch< Nuwber of Medlaeora Mwtc toned by 
K—dgrs In tltim Caeenocies 



Hediacor Category 



Vrlor 



Post 



C-'value 



p-level 



Attitudes 


1.71 


1.06 




Stutot Background 


1.11 


1.39 




CoBBunlty 


0.86 


0.S6 




School Characteristics 


0.32 


0.38 




Classrooa Characteristics 


0.11 


0.23 




Teacher Characteristics 


0.46 


0.66 




Achievement Definition 


0.14 


0.79 


3.23 


Desegregation Definition 


0.96 


1.75 


2.27 


Hcmey 


C.46 


0.23 




Change Frobleas 


0.'17 


0.25 


2.11 


Other 


O.U 


0.57 





.007 
.04 

.06 
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Table 4 

Rdatioaa Batimea Reader's Backgrotrnd. Prior- and Poae-Beliefs About Deaegregafclon^ 



TmIc fteaearch Folitlcal Prior Prior Prior Poat Poat Poae 

Fa«Uiarlty £pertU« Beliefa^ Effect Sl«e^ Confidence Mediators Effect Siae^ Coofide«ce MedUter. 



ople Faailiarity 
m«wdi SKp«rCis« 
•licical Ball«f a 
Hat Iffacc Sin 
wimt Confidma 
vior Madiatora 
mt Bf fact Sisa 
MC CoMfidanca 
OBt Madiatora 



lacaas * •13<p>.03 



-.04 



-.02 
.0? 



-.47* 


,48* 


-.30 


.35 


.38* 


-.24 


-.01 


-.06 


.10 


-.45* 


.00 


.54** 


.01 


.08 


•*e36 


.39 


-.35 


.15 




.12 


.51* 


.01 


-.53* 


.07 






.06 


.48* 


.09 


-.38 








-.12 


-.19 


• 38** 
-.38 1 










-.22 



«. All taata ara baaad on batwaaa 9 and 13 df. 

If ^LltttrvUuaa vara yaad. ona trend e-ergad - r with prior confidence - .45*. 
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Figure 1 

Coding Frawe for Moderators of Descgregattoo Egfects 

I. Attitmdea (support) 

of parents, white, black; of studeats, «rtate. black; of teachers: of school 
adnlnlstratlon; of politicians; of business: of detnr. 

IX. Student background 

parent education; fanily relations; family aixe; social class, blacks, 
whites; achieveMnt/abllity, blacks, whites; age; self concept/personality, 

III. CcwBunity gackgrmnd 

area of country; urban vs. rural; size; historical conditixias; nedia 
presentation. 

IV, School Characteristics 

size; location; quality (currieulua) . 

V, Classrooa Characteristics 

size ( teacher/pup U ratio; I of teachers); seating (interaction) patterns; 
open vs. traditional. 

VI. Teacher Characteristico 

ability; Ub**\l:f^s (expectation) effects; training for desegregatlx)n; race. 

?H. Achievewent Deginition 

neasurement ; subject natter; interpretation of measuri^at. 

VIII. Desegregation Definition 

voluntary vs. forced; conounity involvemenc in impleaencation; need (distance) 
for busing; black/white ratio; length of laplsMntatitm; who gets bused. 

ZZ. Iteiey (expoidltures) 

resources for teachers; other support services (persotmel). 

X. Change of School Probleaa (confusion) 
preparatl<n of students. 

ZZ. Other 



•10 



Six Desegregation Reviews: 37 
Figure 2 

Si^lf leant Eelatlans Between Reader ferceptlons IndivWual 
Keviewa and Reviewer Characteristics 

The quality of a review vaa correlated 

(a) positively with the reader's confidence in its inter- 
pretation (Z - .42, p<.02) 

The persuasiveneas of a rc^ law was correlated 

(a) positively with the reader's judgment of its «iuality 

(2 -1.05, p<.000l) . . ^ 

(b) positively with the reader's confidence in Its inter- 
pretation (Z - .66, p<.00l) 

(c) positively with the reader's Judgment of the number 
of aediators it aentioned (Z «• .32, p<.05) 

The effect size in a review was correlated 

<a) positively with the reviewer's stated effect size 

<2 - .92, p<.000l) - 

Cb) positively with the reviewer's politics (Z - .JZ, 

(c) °nlgatively vith the reviewer's own confidence in their 
laterpretation (Z - -.76, p<.04) g 

(d) negatively with the reader's judgment of the nun&er of 
Mediators it aentioned (Z - -.92, p<.Ol) 

The number of mediators in a review was correlated 

(a) positively with the reviewer's confidence in their 
interpretation (Z - .62. p<.OCK)l) 

(b) positively with the reviewer's politics (Z - .55. p<.oouAj 



Vote: All data are correlational but are repeated under imly 
bei^Ufng for ease of presentation. 
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